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Abstract 


This  thesis  investigates  the  use  of  EEG  to  track  the  spatial  locus  of  covert,  visual 
attention.  Three  experiments  are  described  that  were  to  detect  the  position  of  visual  attention 
as  it  was  deployed  towards  targets  as  they  appeared.  The  first  experiment  uses  flickering  fields 
placed  in  the  periphery  of  the  visual  field  to  induce  SSVEPs,  to  be  used  to  track  the  position  of 
attention  which  varies  horizontally  between  them.  The  flickers  failed  to  produce  significant 
SSVEP  activity.  However  attention  locus  could  still  able  to  be  tracked  by  endogenous 
lateralizations  of  12Hz  and  18Hz  activity.  A  second  experiment  was  then  designed  to  track 
attention  locus  as  it  varied  either  horizontally  or  vertically  using  only  endogenous  EEG  activity  in 
the  alpha  (10Hz),  low-beta  (18Hz),  high-beta  (24Hz)  and  gamma  (36Hz)  bands.  Tracking  proved 
successful  in  all  but  a  small  number  of  subjects.  Horizontally  varying  attention  was  associated 
with  lateralizations  of  the  alpha  band  and  low-beta  band,  while  vertically  varying  attention  was 
associated  with  varying  alpha  band  and  low-beta  band  activity  in  the  occipito-parietal  junction 
over  the  central  sulcus.  A  third  experiment  was  then  performed  to  track  attention  locus  as  it 
varied  in  two  dimensions.  Using  a  combination  of  the  features  found  to  be  informative  in  the 
second  experiment,  tracking  proved  successful  in  up  to  nine  bins  of  two-dimensional  visual 
space.  Tracking  in  either  the  horizontal  or  vertical  dimension  was  also  successful  when 
attention  varied  in  two  dimensions.  The  success  of  this  method  shows  that  EEG  can  be  used  to 
passively  detect  the  spatial  position  of  attention,  at  varying  degrees  of  position,  as  a  person 
attends  to  objects  they  see. 
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I.  Introduction 


Research  into  visual  attention  as  a  control  signal  for  brain-computer  interfaces  (BCIs)  is 
well  into  its  second  decade  (He  et  al.  2013,  Lotte  2005,  Muller-Putz  et  al.  2005,  and  Wolpaw  et 
al.  2002).  Technologies  to  communicate  using  signals  originating  directly  from  brain  activity 
have  been  primarily  targeted  towards  persons  with  debilitating  neurological  conditions  that 
disable  all  muscle  movement  (Birbaumer  et  al.  2008,  Wolpaw  et  al.  2002).  Because  of  this 
particular  population  of  interest,  attention-based  BCI  research  has  relied  upon  users 
manipulating  what  is  referred  to  as  "covert"  attention  or  attention  directed  to  sensory  inputs 
without  the  use  of  orienting  muscle  movements  such  as  saccades  or  head  movements.  There  is 
a  wealth  of  results  on  this  mechanism  of  visual  attention  (Bressler  et  al.  2008,  Corbetta  and 
Shulman  1998  &  2002,  Posner  et  al.  1980,  Shipp  2004,  Wojciulik  and  Kanwisher  2000).  Most 
work  on  attention-mediated  BCIs  has  focused  on  the  visual  modality,  although  there  is  also 
work  on  directed  auditory  (Horton  et  al.  2013,  Horton  etal.  submitted)  and  somatosensory 
attention  (Kaufmann  et  al.  2013,  Muller-Putz  et  al.  2006,  and  Zhang  et  al.  2007). 

BCI  research  on  visual  attention  has  produced  several  methods  for  using  brain  activity 
for  control  (Gerven  and  Jensen  2009,  Lin  et  al.  2006,  Marchetti  et  al.  2013,  Muller-Putz  et  al. 
2005).  BCIs  that  use  visual  attention  for  control  require  either  specialized,  external  stimuli  to 
induce  event-related  potentials  (ERPs)  or  endogenous  brain  activity  associated  with  directed 
covert  visual  attention.  BCIs  that  use  ERPs  have  a  much  longer  history,  and  have  produced  the 
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highest  information  transfer  rates  (ITRs)  to  date  (Bin  et  al.  2009).  However,  the  use  of 
endogenous  brain  signals  by  BCIs  has  advantages  over  methods  that  rely  on  ERPs.  First, 
detecting  endogenous  patterns  of  brain  activity  eliminates  any  obtrusive  visual  stimuli  that  are 
needed  to  induce  ERPs.  Second,  using  the  endogenous  signals  generated  by  mechanisms  of 
covert  attention  lets  users  direct  attention  throughout  the  visual  field  in  a  natural  way.  Third, 
attention  is  innately  associated  with  interest  and  intent,  and  to  what  attention  is  directed  is 
typically  the  signal  of  interest  for  BCIs.  Using  a  BCI  to  detect  what  draws  someone's  attention  is 
far  more  informative  and  useful  than  making  someone  choose  between  predefined  targets. 

BCIs  that  use  endogenous  brain  signals  associated  with  the  direction  of  visual  attention 
were  introduced  by  Kelly  and  colleagues  (2006).  These  systems  use  EEG  or  MEG  to  detect 
lateralization  of  alpha-band  and  beta-band  activity  associated  with  the  orientation  or  position 
of  the  locus  of  attention.  The  strength  of  alpha  band  lateralization  has  been  positively 
correlated  with  both  the  eccentricity  of  the  locus  of  attention  (Bahramisharif  et  al.  2011)  and 
improved  performance  in  visual  discrimination  (Haegens  et  al.  2012).  EEG  BCIs  have  been 
designed  to  use  these  features  for  many  purposes  (Kelly  et  al.  2005,  Ricco  et  al.  2012,  Thorpe  et 
al.  2012,  Tonin  et  al.  2012  &  2013).  MEG-signal  based  BCIs  can  discern  two  spatial  dimensions 
in  the  direction  of  visual  attention  (Bahramisharif  2011,  Gerven  and  Jensen  2009)  and  estimate 
the  polar  angle  of  locus  of  attention  continuously  (Bahramisharif  et  al.  2010).  Though  MEG 
methods  have  been  able  to  out-perform  their  EEG  counterparts  in  degrees  of  freedom  and  ITR, 
EEG  has  obvious  practical  advantages. 
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EEG-based  BCIs  that  use  endogenous  signals  related  to  covert  visual  attention  use  the 


experimental  paradigm  developed  by  Posner  (1980).  This  paradigm  targets  top-down 
mechanisms  that  deploy  attention  to  a  region  of  visual  space  in  which  visual  stimuli  are  absent. 
Assessing  top-down  attention  in  this  way  eliminates  the  influence  of  bottom-up  stimulus-driven 
brain  activity.  EEG  studies  of  this  kind  report  accuracy  rates  in  the  low  80%  range  for  the  best¬ 
performing  subjects  in  one  dimensional,  horizontal,  binary  classification  (Ricco  et  al.  2012, 

Tonin  etal.  2012  &  2013). 

However,  deploying  attention  to  empty  space  is  not  a  natural  behavior,  and  most 
individuals  are  not  adept  at  controlling  top-down  covert,  visual  attention  in  the  way  needed  by 
these  BCIs.  Attention  is  used  to  enhance  incoming  sensory  information  and  is  usually  deployed 
towards  targets  of  interest.  For  covert  visual  attention  BCIs  to  leave  the  laboratory  and  allow 
users  to  transmit  intent  about  the  rich  visual  world  around  them,  a  better  design  would  be  to 
capture  attention  as  it  interacts  with  unaltered,  incoming  visual  stimuli  and  transmit  either 
features  of  said  stimuli  or  the  spatial  locus  of  deployed  attention.  The  goal  of  such  a  BCI  would 
not  be  trial-based  transition  of  a  discrete  choice,  but  the  continuous  tracking  of  the  locus  of 
attention. 

Three  EEG  experiments  will  be  described  that  investigate  both  the  SSVEPs  and 
endogenous  signals  related  to  visual  spatial  attention,  with  the  goal  of  passively  tracking  the 
locus  of  attention.  The  first  experiment  introduces  a  new  way  to  use  flickering  stimuli,  placed 
at  the  periphery  of  the  visual  field,  to  induce  SSVEPs.  The  aim  was  to  use  the  highly-detectable 
SSVEP  while  avoiding  the  drawbacks  of  centrally  placed  flickering  stimuli.  The  experimental 
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hypothesis  was  that  the  magnitude  of  the  SSVEPs  would  be  modulated  by  the  visual  attention 


system  in  a  way  that  depends  on  the  proximity  of  the  covert  attention  locus  to  flickering 
stimuli.  These  locus-dependent  modulations  would  then  be  used  to  passively  track  the  spatial 
location  of  visual  attention. 

The  second  experiment  was  designed  to  determine  whether  the  spatial  locus  of  visual 
attention  can  be  tracked  using  only  endogenous  brain  activity.  The  analysis  sought  to 
characterize  the  endogenous  brain  activity  associated  with  attending  to  visual  targets  that 
varied  in  position  along  a  single  dimension,  either  horizontally  or  vertically,  and  then  to  track 
the  spatial  position  with  predictive  models. 

The  third  experiment,  like  the  second,  was  designed  to  determine  whether  endogenous 
brain  activity  can  be  used  to  track  the  locus  of  visual  attention  passively.  However,  this  time, 
the  positions  of  the  attended  targets  varied  in  two  dimensions.  Tracking  was  then  attempted  in 
either  one  dimension  at  a  time  or  in  two  dimensions  simultaneously. 
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II.  Using  SSVEP  to  track  attention  to  visual  targets  that  vary  in  their 
horizontal  position 

Introduction 

EEG  BCIs  use  one  of  four  brain  signals  to  detect  intent:  slow  cortical  potentials  (SCP); 
event-related  synchrony  or  desynchrony  (ERS/ERD)  associated  with  mental  tasks  like  imagining 
motor  movement  or  directing  covert  visual  spatial  attention;  ERPs  triggered  by  stimuli  like  the 
p300  associated  with  the  appearance  of  an  anticipated  an  anticipated  stimulus,  or  steady-state 
visual  evoked  potentials  (SSVEPs)  induced  by  flickering  visual  stimuli  and  modulated  by  visual 
attention  (Bin  et  al.  2009  and  Riccio  et  al.  2013,  Tonin  et  al  2012  &  2013,  Wolpaw  et  al.  2002, 
Wolpaw  and  McFarland  2004).  The  most  successful  in  terms  of  information  transfer  rate  (ITR) 
have  certainly  been  SSVEP-based  BCIs.  These  systems  typically  report  accuracy  rates  above 
99%  when  distinguishing  between  a  user's  attention  to  one  of  the  discrete,  flashing  targets, 
yielding  ITRs  of  above  60  BPM  (Bin  et  al.,  Burke  et  al.  2005,  Lalor  et  al.  2005,  and  Kelly  et  al. 
2009).  With  such  high  accuracy  and  short  detection  periods,  one  second,  the  limitation  of 
these  systems  is  only  in  user  interface  design,  and  how  many  distinct  targets  can  be  presented. 
Other  methods,  such  as  those  that  use  endogenous  signals  concerning  the  locus  of  attention, 
communicate  at  less  than  half  the  rate.  (Allison  et  al.  2010a). 

The  present  experiment  is  designed  to  use  SSVEPs  in  conjunction  with  the  endogenous 
parietal  and  occipital  alpha  and  beta  band  activities  associated  with  directed  attention  to  track 
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the  locus  of  attention  more  precisely  than  is  possible  when  using  either  modality  alone.  The 


experimental  display  uses  two  checkerboards  that  flicker  at  different  rates.  These 
checkerboards  are  intended  to  generate  SSVEPs,  even  when  the  locus  of  attention  lies  between 
the  two  checkerboards.  This  lets  one  use  EEG  to  estimate  the  locus  of  attention  through  an 
interpolative  procedure.  In  addition  to  this  SSVEP-based  method,  the  separate  alpha  and  beta 
band  lateralizations  associated  with  directed  covert  attention  can  provide  convergent 
information  when  training  an  algorithm  to  predict  the  locus  of  attention. 

Results  indicate  that  the  unattended,  frequency-tagged  stimuli  in  the  periphery  produce 
weak  EEG  signals  and  are  not  useful  in  tracking  the  locus  of  attention.  However,  results  do 
support  the  notion  that  endogenous  brain  activity  associated  with  directed  attention  can  be 
used  to  track  the  horizontal  position  of  the  locus  of  covert  attention  through  an  examination  of 
the  degree  to  which  alpha,  beta,  and  gamma  band  power  is  lateralized. 


Methods 

Experimental  Setup 


Six  male  subjects,  between  the  ages  of  25  and  54,  participated  in  the  experiment.  All 
subjects  had  normal  or  corrected-to-normal  vision,  and  none  reported  possibly  confounding 
neurological  conditions.  Subjects  varied  widely  in  their  experience  with  EEG  and  visual 
attention  experimentation.  Eight  to  ten  sessions  of  EEG  data  collection,  each  roughly  15 


7 


minutes  in  length,  were  performed  by  each  subject  across  either  two  or  three  days.  Work  with 


human  subjects  was  approved  by  the  UC  Irvine  Institutional  Review  Board  under  protocol  2008- 
6342. 


The  experiment  was  conducted  in  an  isolated  room  with  no  light  except  for  that  from 
the  experimental  display.  Subjects  were  seated  at  a  viewing  distance  of  0.76m  from  an  LED 
monitor  of  diagonal  68.6  cm.  The  display  was  positioned  at  eye  level  and  subtended  56°  in 
azimuth  by  26°  in  elevation.  The  display  device  was  controlled  by  Matlab  2012b  (Mathworks) 
and  PsychToolBox  3.0.  EEG  data  was  collected  using  10-20  configuration,  64-sensor, 
Waveguard  EEG  caps.  These  provided  signals  to  an  Advanced  NeuroTechnologies  (ANT)  EEG 
amplifier  at  a  rate  of  256  samples  per  second.  The  output  of  the  amplifier  was  controlled  using 
ANT  libraries  for  Matlab.  During  setup,  all  channels  were  measured  to  have  impedance  below 
10KQ. 


The  display  showed  a  gray  background,  a  white  fixation  cross,  two  peripheral,  contrast- 
reversing  checkerboards,  and  individually  presented  attention  targets  (figure  1).  The  gray 
background,  the  median  allowed  gray  value  in  PsychToolBox,  was  present  at  all  times  during 
the  experiment.  The  white  fixation  cross  was  located  at  the  center  of  the  screen  and  was  2°  in 
width  and  height.  Checkerboards  were  located  at  extreme  right  or  extreme  left  of  the  display, 
subtended  a  rectangle  of  [8°,  27°].  The  checkerboards  were  constructed  of  10  squares  by  30 
squares,  each  with  sides  of  0.89°.  Flickering  field  contrast  reversed  at  either  24  or  36  Hz, 
completing  full  cycles  at  12  or  18Hz,  respectively.  Flickering  fields  were  presented  in  two 
conditions:  1)  18Hz  reversal  at  left  and  12Hz  reversal  at  right,  and  2)  12Hz  reversal  at  left  and 
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18Hz  reversal  at  right.  Single,  black  letters  of  height  1°  were  used  as  attention  targets.  The 
letters  used  were  {A,  F,  H,  L }  and  were  presented  in  uniform-random  sequence  with  no 
sequential  repeats.  Target  positions  varied  horizontally.  Their  vertical  position  was  held 
constant  at  the  center  of  the  display.  Target  horizontal  position  varied  uniformly  within  the 
interval  —17°  and  17°  about  the  center  of  the  display,  which  did  not  overlap  the  regions 
containing  the  flickering  fields.  Target  position  in  any  one  trial  lay  at  least  2°  from  the  previous 
trials. 


Figure  1:  Experimental  display 

Shown  is  the  experiment  display  for  experiment  1.  At  display  left  and  right  are  the  flickering  checkerboards 
stimuli.  In  the  first  condition  18Hz  flicker  was  present  at  left  and  12Hz  flicker  at  right.  In  the  second  condition 
12Hz  flicker  was  present  at  left  and  18Hz  flicker  was  at  right.  At  center,  in  white,  is  the  fixation  cross  to  which 
subjects  directed  their  gaze  during  data  collection.  Letter  H,  to  the  right  of  cross,  is  a  covert  attention  target  for  a 
particular  trial.  Targets  were  presented  at  0°  elevation  and  at  a  horizontal  locations  that  varied  between  —17°  and 
17°  azimuth. 


Subjects  were  instructed  to  hold  their  gaze  at  the  central  fixation  cross  during  data 
collection  and  to  avoid  blinks,  eye  and  muscle  movements.  Each  trial  presented  a  single 
attention  target  in  a  new  horizontal  position  for  1500msec.  Each  trial  was  followed 
immediately  by  the  next.  Subjects  were  instructed  to  fixate  on  the  central  cross  during  each 
trial  while  directing  attention  to  the  target  as  it  appeared.  Subjects  were  further  instructed  to 
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use  the  keyboard  to  indicate  the  identity  of  the  target  letter.  A  total  of  420  trials  were  recorded 


for  each  session,  and  four  sessions  were  recorded  for  both  condition  one  and  condition  two. 


Feature  Extraction 


Further  signal  processing  and  analysis  were  conducted  offline  in  Matlab  and  Python. 

The  first  and  last  trials  of  each  session  were  discarded.  Trials  corrupted  by  blinks  or  saccades 
were  rejected  by  a  method  of  channel  time-series  variance  comparison.  Variance  of  time-series 
from  each  trial  was  computed  at  each  channel.  The  mean  and  variance  of  the  channel  time- 
series  variances  were  computed  within  each  session.  Trials  containing  more  than  five  channels 
with  variance  greater  than  two  standard  deviations  from  the  grand  mean  were  rejected.  Using 
this  method,  between  ten  and  thirty-seven  trials  were  rejected  from  each  session  leaving 
between  383  and  410  trials  per  session  for  the  four  sessions  from  each  subject.  The  two 
mastoid  channels  were  ignored  and  the  remaining  channels  were  filtered  between  1  and  50Hz 
with  a  digital,  band-pass  elliptical  filter.  Spectral  power  was  estimated  by  Morlet  wavelets  at 
the  fundamental  frequency,  and  first  harmonic  frequency  of  the  flickering  fields,  or  12, 18,  24 
and  36  Hz. 

Trial  intervals  of  interest  were  identified  by  t-testing.  Data  from  all  subjects  under  both 
experimental  conditions  were  included.  T-testing  compared  all  wavelet  response  samples,  at 
all  channels.  This  produced  87,544  simultaneous  tests.  The  Bonferonni  method  was  used  to 
correct  for  multiple  comparisons,  and  an  alpha  value  of  0.025  was  used  for  significance  testing. 


10 


Results  of  testing  can  be  seen  topographically  in  figure  2.  Significant,  continuous  group 
separation  was  found  in  parietal,  occipital,  and  frontal  electrodes  at  the  four  frequencies  listed 
above,  between  350  and  600ms  after  the  appearance  of  a  new  attention  target. 


355ms 


418ms 


529ms 


602ms 


Figure  2:  Feature  discovery  by  t-testing 

EEG  channels  that  help  to  distinguish  left  vs.  right  locus  of  attention  between  355  (leftmost  topographic  plot)  and 
602ms  (right-most)  after  target  appearance  are  revealed  by  statistical  results.  Displayed  are  normalized  t-values 
from  testing  of  the  low-beta  band  (16-20Hz)  at  352,  418,  527  and  602ms  after  target  appearance.  This  period 
displayed  the  longest,  continuous,  significant  group  separation  during  the  trial  period.  For  topographic 
visualization,  corrected  t-values  were  normalized  to  display  statistically  significant  differences  when  values  are  oft 
are  <-l  or  >1. 


To  generate  a  Common  Spatial  Pattern  (CSP)  feature  set,  wavelet  responses  at  12, 18, 
24,  36Hz  were  integrated  between  350  and  600ms  post  target  appearance.  Data  processing 
thus  resulted  in  248  (4  frequency  power  estimates  at  62  channels)  CSP  features  per  trial,  which 
were  then  centered  and  sphered  within  session  in  preparation  for  tracking  algorithm  training 
and  testing  (n  «  1990,  d  =  248). 
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Predictive  modeling 


Cross-validated  classification  was  conducted  with  linear  and  nonlinear  predictive  models 
to  estimate  the  precision  and  reliability  by  which  locus  of  attention  can  be  predicted  in  the 
presence  of  peripheral  flicker.  Bayes-optimal,  Discriminant  Analysis  and  Random  Forests  (RF) 
were  used.  Prior  to  cross-validation  block  data-splitting,  outliers  were  removed  by  nearest 
neighbor  data-prototyping  using  the  5-metric  described  by  Harmeling  and  colleagues  (2006). 
Discriminant  analysis  used  a  spherical  Gaussian  prior  and  shared  covariance  structure,  yielding 
a  linear  discriminant  (LDA)  and  regularized  MAP  solution.  Random  Forests  included  ensembles 
of  55  boosted,  bagged  and  pruned  decision  trees.  The  optimal  number  of  trees  was  chosen  by 
cross-validation.  Nine-fold  cross-validation  was  used  and  classification  results  are  reported  as 
the  empirical,  binomial  distribution  (expected  and  97.5%  central  density)  of  accurate  attention 
location  prediction.  For  classification  of  attention  location,  the  34.48°  sampling  space  of 
attention  targets  was  divided  into  2, 3, 5  or  7  equal-sized  bins  of  17.24°,  11.49°,  6.90°  and 
4.93°  respectively  (figure  3).  Random  Forest  training  included  permutation  feature  importance 
to  investigate  spatial-spectral  contribution  to  predictability  of  locus  of  attention.  LDA  solutions 
used  for  topographic  visualization  were  fit  with  all  post-outlier  removal  available  data. 
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Figure  3:  Binning  of  attention  target  space  for  n-way  classification 

To  determine  predictive  accuracy  at  increasing  precision,  the  attention  sampling  space  was  divided  into  2,  3,  5  or  7 
horizontal  bins  (subfigure  a,  b,  c  and  d  respectively). 


Results 

Figure  4  shows  the  accuracy  results  when  predicting  attention  location  from  EEG  using 
LDA  and  Random  Forests.  For  conditions  1  and  2,  locus  of  attention  could  be  accurately 
predicted  with  LDA,  at  a  rate  significantly  greater  than  chance,  when  the  target  space  was 
divided  into  2,  3,  5  or  7  bins.  With  the  exception  of  two  subjects,  for  all  test  conditions,  the 
97.5%  central  density  of  the  empirically-derived,  classification  accuracy  distribution  did  not 
overlap  with  the  chance  classification  rates.  Similarly,  nonlinear  Random  Forest  models  yielded 
classification  accuracy  above  chance,  under  the  same  conditions,  though  performance  was 
systematically  lower  than  corresponding  linear  models.  In  terms  of  communication  rate,  a 
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maximal  ITR  of  14.22  BPM  was  found  for  subject  4,  in  the  three-bin  case,  with  an  accuracy  of 


55%. 
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Figure  4:  Attention  tracking  accuracy  rates  at  increased  discriminant  precision 

a,b:  Empirical,  expected  predictive  accuracy  and  97.5%  Cl  for  LDA  models  as  attention  target  sampling  space  is 
divided  into  more  numerous  and  narrow  bins.  In  bold  black  is  the  chance  accuracy  rate  at  each  binning.  All 
tracking  rates  can  be  seen  to  be  significantly  better  than  chance  except  for  the  rates  of  subject  5. 
c,d:  Empirical  expected  accuracy  and  97.5%  Cl  of  Random  Forest  models.  Performance  is  systematically,  but  not 
significantly,  lower  than  that  found  with  linear  models  with  the  exception  of  subject  4,  condition  2  with  3  bins. 
Tracking  performed  with  better  than  chance  accuracy  in  all  but  subjects  3  and  5. 


Inspection  of  confusion  matrices  from  the  LDA  models  indicates  notable  patterns  in 
errors  common  to  high  and  low  performing  subjects  (figures  5  &  6).  Across  subjects  and 
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analysis  conditions,  the  central-most  bins  were  always  predicted  with  the  most  accuracy, 
followed  by  extreme  left  and  extreme  right  bins.  Results  of  this  type  imply  that  the  most 
identifiable  spectral-spatial  EEG  features  are  produced  when  a  subject's  covert  attention  locus 
is  either  very  near  or  very  distant  from  the  location  of  gaze.  Furthermore,  high-performing 
subjects  display  confusion  matrices  with  block-diagonal  patterns,  indicating  that  when 
prediction  errors  are  made,  the  incorrectly  predicted  bin  is  adjoining  the  true  location  of  the 
subject's  attention.  Lower-performing  subjects  did  not  exhibit  any  of  these  patterns  in  their 
confusion  matrices. 
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Figure  5:  Confusion  matrices  from  the  best  (a,c)  and  worst  (b,d)  performing  subjects  in  condition  1 

The  confusion  matrices  of  the  LDA  models  indicate  the  probability  of  a  model  to  predict  one  of  the  n-bins  given  the 
actual  attention  location  in  the  held-out  data.  Actual  attention  location  bins  are  ordered  along  the  rows  from  top, 
the  leftmost  attention  bin,  to  bottom,  the  rightmost  attention  bin.  Error  free  models  would  display  a  perfectly 
diagonal  matrix  structure.  Despite  the  LDA  models  not  being  designed  with  target  spatial  order  in  mind  during 
fitting,  the  best  performing  subjects'  confusion  matrices  display  block-diagonal  structure.  This  indicates  that 
models  fit  to  subjects  with  the  highest  accuracy  rates  most  often  make  prediction  errors  into  bins  neighboring  the 
correct  attention  location.  Such  a  result  implies  shifting  locus  of  attention  and  informative  alpha  and  beta  band 
power  EEG  features  covary  in  a  linearly  consistent  fashion  in  high-performing  subjects, 
a:  The  5-bin  LDA  model  of  the  best  performing  subject, 
b:  The  5-bin  LDA  model  of  the  worst  performing  subject, 
c:  The  7-bin  LDA  model  of  the  best  performing  subject, 
d:  The  7-bin  model  of  the  worst  performing  subject. 
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Figure  6:  Confusion  matrices  from  the  best  (a,c)  and  worst  (b,d)  performing  subjects  in  condition  2 

As  in  figure  5,  models  from  the  best  performing  subjects,  fit  under  condition  two  also  displayed  block-diagonal 

structure,  while  models  fit  to  the  poorest  performing  subjects  error  structure  was  uniform. 

a:  The  5-bin  LDA  model  of  the  best  performing  subject. 

b:  The  5-bin  LDA  model  of  the  worst  performing  subject. 

c:  The  7-bin  LDA  model  of  the  best  performing  subject. 

d:  The  7-bin  model  of  the  worst  performing  subject. 


Figures  7  and  8  show  the  linear  models  of  the  best  performer  for  conditions  1  and  2, 
respectively.  These  subjects  display  a  relative  decrease  in  alpha,  low-beta,  and  gamma  band 
synchrony  in  parietal,  temporal,  and  occipital  regions  contralateral  to  location  of  attention  (for 
attention-left  and  attention-right  trials).  Slight  synchrony  in  frontal-eye-field  regions  can  also 
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be  seen  for  attention-left  and  attention-right  trials.  For  trials  with  covert  attention  to  the 
central  bin,  near  to  fixation,  a  relative  decrease  in  power  at  these  bands  in  all  of  these  regions 
can  be  seen.  Random  Forest  classifier  performance  relied  primarily  upon  lateral-parietal  and 
temporal  power  in  the  alpha  and  low-beta  bands.  These  activity  patterns  were  less  spatially 
and  spectrally  specific,  or  completely  absent  in  the  subjects  with  chance  or  near  chance 
performance  (figures  9  &  10).  These  spectral-spatial  feature  patterns  preferred  by  good 
performing  models  are  consistent  with  the  covert  visual  spatial  attention  "orienting  network" 
described  by  Posner  and  colleagues  (2007). 

Predictive  performance  and  model  coefficients  were  essentially  the  same  across 
conditions  1  and  2.  This  is  not  consistent  with  the  experimental  hypothesis  that  peripheral 
flicker  induces  bottom-up  SSVEP  activity. 
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Figure  7:  Models  from  the  best  performing  subject  in  condition  1 

Left:  LDA  model  class  means,  contralateral  desynchrony  in  alpha,  low  and  high-beta  and  gamma  bands  can  be  seen 
in  visual,  parietal  and  temporal  regions.  Desynchrony  is  seen  bilaterally  when  covert  attention  location  is  near  to 
fixation. 

Right:  RF  feature  importance  indicates  performance  depends  primarily  on  features  from  the  alpha  and  low-beta 
band  in  occipital  and  parietal-occipital-temporal  junction. 
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Figure  8:  Models  from  the  best  performing  subject  in  condition  2 

Left:  LDA  model  class  means  for  subject  4,  condition  2,  with  3  bins.  Spectral-spatial  patterns  are  similar  to  those 
seen  in  condition  1.  Unattended  visual  stimuli  appear  to  have  little  effect  on  the  EEG  power  at  frequencies  and 
harmonics  of  the  stimulus  flicker  rate  while  directing  attention. 

Right:  Alpha  and  low-beta  activity  in  occipital  and  parietal-occipital  junction  contribute  most  to  accurate  tracking 
of  attention.  The  change  in  position  of  the  flickering  fields  has  no  effect  on  RF  model  feature  importance. 
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Figure  9:  Models  from  the  worst  performing  subject  in  condition  1 

Left:  LDA  model  class  means  for  the  worst  performing  subject,  subject  5,  condition  1,  with  3  bins.  Contralateral 
desynchrony  prominent  in  high  performing  subjects  is  absent. 
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Right:  RF  Feature  importance  shows  little  spectral  or  spatial  specificity  from  models 
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Figure  10:  Models  from  the  worst  Performing  subject  in  condition  2 

Left:  LDA  model  class  means  for  subject  5,  condition  1,  with  3  bins.  None  of  the  distinct  synchrony  seen  in  high 
performing  subjects  is  present. 

Right:  RF  feature  importance  also  shows  little  spectral  or  spatial  specificity  from  common  in  models  from  better 
performing  subjects. 
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Discussion 


Peripheral  flickering  stimuli  had  no  effect  on  either  activation  patterns  or  classification 
results.  Despite  this  negative  result,  tracking  accuracy  for  most  subjects  was  still  at  rates 
significantly  above  chance.  Classification  rates,  for  both  conditions,  in  the  two-bin  case  are 
similar  to  those  seen  in  offline  and  online  covert  visual  spatial  attention  based  EEG  BCI  systems 
(Tonin  et  al.  2012  &  2013).  Reported  accuracy  in  these  studies  was  between  65  and  70%  for 
most  subjects.  Detection  algorithms  in  these  previous  studies  utilize  a  time-dependent, 
integrated  evidence  method  and  work  with  features  from  the  alpha  band  only.  The  CSP 
approach  presented  here  uses  multiple  canonical  frequency  bands  and  performs  comparably 
offline.  Previous  studies  direct  a  subject's  attention  to  one  of  two  specific  attention  locations, 
either  to  left  or  to  right,  in  anticipation  of  an  oncoming  target,  and  analyze  the  ISI  period.  This 
covert  visual  spatial  attention  method  not  only  tracks  attention  over  the  continuous  visual  field, 
but  does  so  when  attention  is  actively  engaged;  a  more  natural  behavior.  Further  significant 
predictive  performance  in  the  three  through  seven  bin  cases  is  a  new  result. 

Regions  implicated  in  predictive  modeling  are  consistent  with  previous  covert  visual 
spatial  attention  EEG  studies  (Rihs  et  al  2007,  Thorpe  et  al.  2012,  and  Sauseng  et  al.  2005),  MEG 
(Bahramisharif  et  al.  2010  and  Gerven  et  al.  2009)  and  fMRI  studies  (Silver  2005).  These  include 
frontal-eye-field  (FEF),  occipital  and  parietal  cortex,  specifically  intraparietal  sulcus  (IPS). 
Directing  covert  attention  away  from  the  center  of  gaze  is  associated  with  marked  desynchrony 
in  contralateral  parietal  regions,  perhaps  caused  by  increased  non-synchronous  activity  in  these 
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regions.  Directing  covert  visual  spatial  attention  towards  gaze  location  is  more  associated  with 


desynchrony  in  parietal  lobes  of  both  hemispheres.  Parallel  activity  in  both  parietal  lobes  was 
also  reported  by  Silver  (Silver  2005).  Novel  to  these  results  are  the  patterns  of  activity  in  low- 
beta  and  gamma  bands  in  the  above-mentioned  regions.  Previous  studies  focus  on  variations 
of  activity  only  in  the  alpha  band.  Results  from  this  study  indicate  that  low-beta  is  equally 
important  when  tracking  locus  of  covert  visual  spatial  attention. 

The  null-effect  of  peripheral,  unattended  flicker  on  either  classification  or  model  fitting 
results  could  be  the  explained  in  one  of  two  ways:  1)  the  flicker  introduces  symmetric  noise  into 
regions  and  EEG  bands  useful  in  tracking  covert  visual  spatial  attention  or  2)  the  attention 
system  effectively  suppresses  these  very  conspicuous,  unattended  stimuli.  This  could  be  solved 
by  repeating  this  experiment  without  flicker,  and  comparing  prediction  rates.  If  prediction 
rates  are  comparable,  and  potentially  influential  stimuli  such  as  these  are  effectively 
suppressed  as  far  as  this  attention  tracking  method  is  concerned,  then  it  is  plausible  that 
tracking  covert  visual  spatial  attention  could  be  performed  as  effectively  in  a  much  more 
visually-rich  scene.  This  would  motivate  further  research  using  this  attention  tracking  method 
in  a  real  world  or  virtual  environment. 
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III.  Tracking  attention  in  one  dimension  using  EEG 


Introduction 

EEG  BCIs  using  endogenous  brain  signals  resulting  from  directed  attention  have  several 
advantages  over  ERP  based  methods.  First  is  that  there  is  no  need  to  display  extraneous  visual 
stimuli  in  order  to  evoke  SSVEPs.  A  second  is  that  a  method  that  can  passively  track  the  locus  of 
attention  is  likely  to  be  more  natural  and  intuitive  to  the  user. 

The  results  from  the  first  experiment  indicate  that  it  is  feasible  to  track  attention  using 
CSP  features  generated  from  canonical  EEG  power-bands.  Furthermore,  using  these  features, 
attention  locus  can  be  tracked  in  up  to  seven  distinct  bins  of  horizontal  visual  space.  The 
present  experiment  and  the  experiment  to  be  described  in  the  next  chapter  were  then  devised 
to  further  investigate  this  finding  by  conducting  experiments  similar  to  experiment  one,  but 
with  the  flickering  fields  removed. 

The  aim  of  the  present  study  was  to  infer  EEG  power-band  features  associated  with 
changes  in  position  of  attention  locus  along  azimuth  and  along  elevation.  In  addition, 
predictive  modeling  was  used  to  determine  the  spatial  precision  with  which  visual  attention  can 
be  tracked  using  informative  EEG  features.  Results  showed  that  endogenous  EEG  power-band 
features  can  be  used  to  track  visual  attention  horizontally  or  vertically.  Furthermore,  attention 
can  be  tracked  with  greater  spatial  precision  than  has  yet  been  reported. 
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Methods 


The  EEG  experiment  consisted  of  two  conditions,  one  where  the  subject's  locus  of 
attention  location  varied  in  the  horizontal  direction  only  and  one  where  the  subject's  locus  of 
attention  varied  in  the  vertical  direction  only.  Informative  EEG  power-band  features  were  then 
detected  by  t-testing.  Linear  and  nonlinear  classification  methods  were  then  applied  to 
evaluate  the  feasibility  of  attention  tracking. 

11  subjects  (ten  male  and  one  female,  age  range  21-54)  participated  in  the  experiment. 
All  subjects  had  normal  or  corrected-to-normal  vision,  and  none  reported  possibly  confounding 
neurological  conditions.  Subjects  varied  widely  in  their  experience  with  EEG  and  visual 
attention  experimentation.  Twenty  sessions  of  EEG  data  collection,  each  roughly  fifteen 
minutes  in  length,  were  performed  by  each  subject  across  either  two  or  three  days.  Work  with 
human  subjects  was  approved  by  the  UC  Irvine  Institutional  Review  Board  under  protocol  2008- 
6342. 


The  experiment  was  conducted  in  an  isolated  room  with  no  light  except  for  that  from 
the  experimental  display.  Subjects  were  seated  at  a  viewing  distance  of  0.76  from  an  LED 
monitor  of  diagonal  68.6  cm.  The  display  was  positioned  at  eye  level  and  subtended  56° 
azimuth  by  26°  elevation  and  was  controlled  by  Matlab  2012b  (Mathworks)  and  PsychToolBox 
3.0.  EEG  data  was  collected  using  gelled  electrode,  international  10-20  configuration,  64-sensor 
Waveguard  EEG  caps.  These  provided  signals  to  an  Advanced  NeuroTechnologies  (ANT)  EEG 
amplifier  at  a  rate  of  256  samples  per  second.  The  output  of  the  amplifier  was  controlled  using 


27 


ANT  libraries  for  Matlab.  During  setup,  all  channels  were  measured  to  have  impedance  below 


10KQ. 


The  experimental  display  showed  a  gray  background,  a  white  fixation  cross,  and 
individually-presented  attention  targets  in  the  form  of  black  letters.  The  white  fixation  cross 
was  located  in  the  center  of  the  experimental  display  and  subtended  2°  of  visual  angle.  The 
letters  used  as  attention  targets  were  1°  of  visual  angle  in  height  and  were  uniformly  sampled 
from  the  set  {A,F,H,L}. 

The  experiment  had  two  conditions.  In  the  first  condition,  attention  targets  were 
presented  at  0°  of  elevation  and  varied  in  horizontal  position  between  -17°  (left)  and  17° 
(right).  In  the  second  condition,  attention  targets  were  presented  at  0°  of  azimuth  and  varied 
vertically  in  position  between  -17°  (down)  and  17°  (up).  Sequential  target  positions  were  a 
minimum  of  2°  of  visual  angle  from  the  preceding  target  and  target  letters  did  not  repeat. 

Subjects  were  instructed  to  hold  their  gaze  at  the  central  fixation  cross  during  data 
collection  and  to  avoid  blinks,  saccades  and  other  muscle  movements.  Each  trial  presented  a 
single  attention  target  in  a  new  position  for  1500msec,  and  was  followed  immediately  by  the 
next  trial.  While  maintaining  fixation,  subjects  directed  their  attention  towards  the  newly- 
presented  target,  maintained  attention  upon  the  target,  and  responded  with  a  key-press 
indicating  the  identity  of  the  letter.  A  total  of  420  trials  were  recorded  for  each  session.  Up  to 
ten  sessions  were  collected  on  a  given  day.  Each  session  consisted  of  only  one  experimental 
condition.  The  session  type  was  randomized  and  four  sessions  of  data  were  conducted  for 
conditions  one  (horizontal)  and  two  (vertical). 
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Feature  Discovery 


Orienting  visual  attention  to  the  left  or  right  of  fixation  is  known  to  be  associated  with 
lateralized  changes  in  alpha,  beta  and  gamma  band  EEG  activity  (Bahramisharif  et  al.  2010  & 
2011,  Thorpe  et  al.  2012).  Results  from  experiment  one  corroborated  these  findings.  However, 
a  data-driven  approach  to  discovering  features  useful  for  tracking  attention  was  used  for  this 
experiment.  This  was  done  to  both  reaffirm  the  importance  of  these  attention-associated 
features,  in  this  dataset,  and  to  discover  any  new,  informative  EEG  power  band  features  useful 
in  tracking  the  locus  of  attention. 

EEG  data  were  normalized  by  average  reference  by  the  amplifier  during  recording. 
Further  signal  processing  and  analysis  were  conducted  offline  using  Matlab  and  Python.  The 
first  and  last  trials  of  each  session  were  discarded  outright.  Trials  corrupted  by  blinks,  saccades 
or  other  muscle  artifact  were  rejected  by  a  method  of  channel  time-series  variance  comparison. 
The  variance  of  EEG  channel's  time-series  was  computed  for  each  trial.  The  mean  and  standard 
deviation  of  channel  time-series  variances  were  then  computed  within  each  session.  Trials 
containing  five  or  more  channel  variances  greater  than  two  standard  deviations  from  the  grand 
variance  mean  were  then  rejected  from  further  analysis.  This  method  rejected  anywhere  from 
eight  to  thirty-five  trials  per  session.  The  two  mastoid  channels  were  then  removed  from  the 
dataset.  The  remaining  channels  were  filtered  between  1  and  50Hz  with  a  digital,  band-pass 
elliptical  filter.  Spectral  power  was  estimated  in  a  time-varying  fashion  using  Morlet  wavelets 
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(Schiff  1994)  centered  at  4,  6, 10, 18,  24  and  36  Hz  each  within  the  delta,  theta,  alpha,  low-beta, 


high-beta  and  gamma  bands,  respectively,  and  was  then  down-sampled  by  a  factor  of  four. 

Power-band  feature  discovery  was  conducted  by  massive  t-testing  of  spectral  power 
data  from  all  subjects  in  either  the  horizontal  and  vertical  conditions.  For  the  horizontal 
condition,  groups  for  t-testing  were  defined  as  trials  in  which  attention  was  directed  to  either 
the  left  or  the  right  of  fixation.  For  vertical  condition,  groups  were  defined  as  trials  where 
attention  was  directed  above  or  below  fixation.  T-testing  compared  all  wavelet  response 
samples,  at  all  channels.  This  produced  131316  simultaneous  tests.  The  Bonferonni  method 
was  used  to  correct  for  multiple  comparisons,  and  an  alpha  value  of  0.025  was  used  for 
significance  testing.  Results  of  testing  can  be  seen  topographically  in  figure  11  and  12. 
Continuous,  significant  group  differences  were  found  in  alpha,  low-beta,  high-beta  and  gamma 
band  of  the  parietal  and  occipital  regions  between  350  and  600ms  after  the  appearance  of 
attention  targets.  These  spatial  and  spectral  features  match  well  those  described  in  experiment 
one  and  published  studies  (Bahramisharif  et  al.  2011,  Cosmelli  et  al.  2011,  Thorpe  et  al.  2012). 
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Figure  11:  Feature  discovery  by  t-testing,  horizontal  condition 

The  maximal  differences  between  directing  covert  attention  to  the  left  vs.  to  the  right  was  found  in  EEG  alpha,  low- 
beta,  high-beta  and  gamma  band  power  over  parietal  and  occipital  regions  roughly  between  350  and  600ms  after 
attention  target  appearance.  This  interval  was  found  empirically  by  Bonferonni-corrected  t-testing,  in  the  two- 
binned  (figure  13),  horizontal  condition  data  from  all  subjects.  Trial  periods  prior  to  and  after  this  interval 
displayed  only  sporadic  significant  differences.  Features  used  for  predictive  analysis  were  extracted  from  these 
EEG  power  bands  during  this  time  period.  Displayed  are  normalized  t-values  from  testing  of  the  low-beta  band 
(16-20Hz)  at  352,  418,  527  and  602ms  after  target  appearance.  For  topographic  visualization,  corrected  t-values 
were  normalized  to  display  statistically  significant  differences  when  values  are  oft  are  <-l  or  >1. 
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Figure  12:  Feature  discovery  by  t-testing,  vertical  condition 

Displayed  are  the  t-testing  results  of  the  low-beta  band  for  the  vertical  condition  using  a  two-way  binning  of  the 
vertical  attention  location  sample  space  (figure  13).  The  comparison  was  made  between  time-varying  low-beta 
power  when  attention  was  directed  above  fixation  minus  the  time-varying  low-beta  power  when  attention  was 
directed  below  fixation.  Red  values  indicate  more  low-beta  EEG  power  when  attending  upward,  while  blue  values 
indicate  more  low-beta  EEG  power  when  attending  downward.  Like  in  figure  3,  corrected  t-values  were 
normalized  to  display  statistically  significant  differences  when  values  are  oft  are  <-l  and  >1. _ 


Further  analysis  was  restricted  to  data  from  the  interval  [350,600]  msec  concerning  the 
power  in  the  informative  bands  alpha,  low-beta,  high-beta  and  gamma.  To  generate  a  Common 
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Spatial  Pattern  (CSP)  feature  set,  wavelet  responses  in  the  alpha,  low-beta,  high-beta  and 
gamma  bands  were  integrated  between  350  and  600msec  post  target  appearance.  Data 
processing  thus  resulted  in  a  248  (4  frequency  power  estimates  at  62  channels)  CSP  features 
per  trial,  which  were  centered  and  sphered  within  session  in  preparation  for  tracking  algorithm 
training  and  testing  (n  =  1532+  for  horizontal  and  vertical  condition  and  n  =  4596+  for  the  two 
dimensional  condition,  d  =  248). 


Predictive  Modeling 


To  estimate  the  precision  and  reliability  with  which  the  locus  of  attention  can  be 
tracked,  predictive  modeling  was  conducted  with  linear  and  nonlinear  classifiers.  Linear  Bayes- 
optimal  Discriminant  Analysis  and  Random  Forest  models,  were  chosen,  as  in  experiment  one. 
Outliers  were  removed  by  nearest  neighbor  data-prototyping  using  the  5-metric  described  by 
Harmeling  and  colleagues  (2006).  Discriminant  analysis  used  a  spherical  Gaussian  prior  and 
shared  covariance  structure  yielding  a  regularized  MAP  solution  and  linear  class  boundaries. 
Random  Forests  included  ensembles  of  55  boosted,  bagged  and  pruned  decision  trees.  Optimal 
numbers  of  trees  were  established  using  subsets  of  training  data  during  cross-validation.  Nine¬ 
fold  cross-validation  was  used  and  classification  results  are  reported  as  the  empirical,  binomial 
distribution  (expected  and  97.5%  central  density)  of  accurate  attention  location  prediction.  For 
prediction  of  attention  location  in  either  the  horizontal  or  the  vertical  condition,  the  34° 
sampling  space  of  attention  targets  was  divided  into  two,  three,  five  or  seven  equal-sized  bins 
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of  17.24°,  11.49°,  6.90°  and  4.93°,  respectively  (figure  13).  LDA  solutions  used  for  topographic 


visualization  were  fit  with  all  data  after  cross-validated  testing  was  complete.  During  cross- 
validation  testing,  Random  Forest  training  included  recording  of  permutation  feature 
importance  to  investigate  spatial-spectral  feature  contribution  to  predictability  of  locus  of 
attention.  Confusion  matrices  were  computed  during  cross-validation  testing  to  visualize  the 
distribution  of  correct  and  incorrect  predictions  in  the  test  data. 

Communication  speed  was  computed  using  ITR,  a  measure  using  the  probability  of 
successful  communication  of  a  message,  the  time  taken  to  pass  the  message  and  the  number  of 
discrete  messages.  ITR  is  computed  as 

ITR  =  (log2  n  +  p  *  log2  p  +  (1  -  p)  *  log2  \ , 

where  the  units  of  ITR  is  bits-per-minute  (BPM),  n  is  the  number  of  different  messages,  p 
is  the  probability  of  successfully  passing  the  message,  and  t  is  the  time  in  minutes  to  pass  the 
message.  For  this  experiment,  n  was  the  variable  number  of  bins  into  which  the  attention 
sampling  space  was  divided,  and  p  was  the  empirical,  expected  rate  of  successful  tracking.  The 
CSPs  used  required  600msec,  or  0.01  min.,  of  EEG  data  to  generate  features  while  trained 
models  would  require  an  insignificant  amount  of  time  to  make  a  decision.  Therefore,  0.01  min. 
was  used  as  t  to  compute  ITR. 
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Figure  13:  Binning  of  attention  target  space  for  n-way  classification 

The  experiment  display  is  shown  with  several  overlays  indicating  the  spatial  binnings  used  for  tracking.  The 
experiment  environment  consisted  of  a  gray  background,  a  white  fixation  cross  and  individually  presented 
attention  targets.  Attention  targets  took  the  form  of  black  letters  (the  letter  H  in  this  figure). 

Left:  Binning  of  horizontally  varying  attention  targets  from  two  to  seven  bins 
Right:  Binning  of  vertically  varying  attention  targets  from  two  to  seven  bins 
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Results 


Horizontal  Tracking 


Tracking  of  the  locus  of  attention  from  EEG  data  was  successful  in  almost  all  cases. 
Results  when  tracking  horizontally  varying  attention  into  two,  three,  five  and  seven  bins  can  be 
seen  in  figure  14.  These  results  show  that  the  locus  of  attention  could  be  predicted  at  a  rate 
significantly  above  chance  for  nine  of  the  11  subjects.  Put  more  precisely,  the  0.0125% 
experimental  lower-bound  binomial  posterior  probability  of  successful  classification  was  above 
chance  in  these  nine  subjects.  Furthermore,  in  five  of  the  11  subjects,  the  posterior  success 
lower  bound  was  at  least  15%  better  than  chance  in  two-way  to  seven-way  classification.  The 
best-performing  subjects  displayed  expected  accuracies  of  68.0%  in  two-way,  51.1%  in  three- 
way,  30.5%  in  five-way  and  24.2%  in  seven-way  classifications  as  compared  to  chance  accuracy 
rates  of  50.0%,  33.3%,  25.0%  and  14.3%,  respectively.  Regularized  linear  models  generally 
performed  better  than  nonlinear  models.  Though  this  difference  was  consistent  across  subjects 
and  granularities  of  tracking,  the  posterior  probabilities  of  success  did  not  diverge  in  a 
statistically  significant  fashion. 
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Figure  14:  Tracking  accuracy  rates  from  the  horizontal  condition  at  increased  discriminant  precision 

Empirical,  expected  predictive  accuracy  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear 
Random  Forest  models  (bottom)  as  the  horizontal  attention  location  was  tracked  with  greater  granularity  (binning 
shown  in  figure  13).  Chance  accuracy  at  each  number  of  bins  displayed  as  the  bold,  black  line.  All  but  a  few 
subjects  performed  significantly  better  than  chance  up  to  and  including  7  horizontal  bins.  Linear  models,  in  most 
cases,  performed  better  than  nonlinear  models. 


The  communication  speed  and  the  empirical  97.5%  central  density  resulting  from 
horizontal  tracking  using  linear  and  nonlinear  models  can  be  seen  in  figure  15.  In  many  cases 
the  high  tracking  accuracy  rate  of  the  two-bin  case  produces  the  highest  expected  ITR.  The 
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highest  ITR  seen  in  this  experiment  is,  in  fact,  the  two-bin  case  for  subject  10,  which  is  11.35 


BPM.  However,  there  are  several  subjects  with  maximal  ITR  in  the  three  bin  case.  In  these 
cases,  the  reduced  tracking  accuracy  is  offset  by  the  increased  number  of  target  bins  among 
which  discrimination  is  being  made. 
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Figure  15:  ITRs  from  the  horizontal  condition  at  increased  discriminant  precision 

Empirical,  expected  ITRs  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear  Random  Forest 
models  (bottom)  as  the  horizontal  attention  target  sample  space  was  divided  into  more  numerous  and  narrow  bins 
(as  shown  in  figure  1).  The  maximal  ITR  is  seen  for  subject  10  in  the  two-bin  case,  where  only  hemifield  of 
attention  is  being  detected.  However,  for  several  other  subjects  the  three-bin  case  yields  the  highest  ITR.  ITR  is 
greater  in  these  cases  because  the  reduction  in  classification  accuracy  is  offset  by  the  greater  number  of  horizontal 
spatial  bins  among  which  attention  locus  is  being  discriminated. 


The  models  of  the  best-performing  and  worst-performing  subjects,  when  tracking 
attention  into  three  horizontal  bins,  are  displayed  in  figures  16  and  17,  respectively.  Linear 
model  class  means  show  that  relatively  more  ipsilateral  and  less  contralateral  alpha,  low-beta 
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and  high-beta  power  is  seen  in  parietal  and  occipital  regions  when  attention  is  directed  to  the 


left  or  right.  When  attention  is  directed  near  fixation,  relatively  less  EEG  power  is  seen  in  these 
bands  bilaterally  in  parietal  and  occipital  regions.  Less  dramatic  differences  in  gamma  band 
power  are  seen  over  the  entire  head.  Random  Forest  feature  importance  indicates  that  alpha, 
low-beta  and  high-beta  parietal  and  occipital  features  contributing  most  to  correct  tracking. 
These  patterns  of  spatial  power-band  activity  were  common  amongst  all  good  performing 
subjects,  while  poorer  performing  subjects  did  not  display  these  concentrated  differences  in 
activity  (figure  17). 
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Left:  LDA  model  class  means  for  the  3  bin  case.  Class  means  are,  from  left  to  right,  attention  left, 
attention  ahead  and  attention  right.  Increased  synchrony  can  be  seen  in  red  while  desynchrony  can  be 
seen  in  blue.  Attention  directed  to  the  left  or  right  is  associated  with  increased  synchrony  ipsilaterally  and 
desynchrony  contralaterally  in  alpha,  low-beta  and  high-beta  bands  of  parietal  and  occipital  cortical 
regions.  Attention  directed  near  fixation  is  associated  with  desynchrony  in  both  hemispheres.  These 
results  are  typical  of  high  performing  subjects. 

Right:  RF  feature  importance  indicates  that  predictive  performance  depends  primarily  on  features  from 
the  alpha,  low-beta  and  high-beta  bands.  These  features  correspond  to  the  notable  synchrony  and 
desynchrony  in  the  LDA  models. 
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Figure  17:  Models  from  the  worst-performing  subject  (5)  in  the  horizontal  condition 

Models  fit  to  data  from  subjects  with  poorer  predictive  performance  do  not  display  the  characteristic 
ipsilateral  synchrony  and  contralateral  desynchrony  associated  with  horizontally  directed  covert  visual 
attention. 
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Confusion  matrices  of  the  best-performing  subjects  displayed  a  near-block-diagonal 


structure  (figure  18).  Rows  of  the  confusion  matrices  indicate  the  true  attention  location,  from 
top  row  to  bottom  row  ordered  from  attention  left  to  attention  right.  The  entries  along  the 
columns  indicate  the  probability  of  predicting  any  attention  location  bin  given  the  true  location 
of  attention.  Perfect  classification  upon  testing  data  would  display  pure  diagonal  matrices,  with 
all  probabilities  of  prediction  of  a  particular  attention  location  into  the  correct  bin.  The  near¬ 
block-diagonal  structure  of  the  matrices  indicates  that  when  the  models  are  making  incorrect 
predictions,  they  are  most  often  predicting  into  a  bin  neighboring  the  correct  bin.  This  result 
was  serendipitous  as  the  proximity  of  the  attention  bins  was  not  built  into  model  optimization. 
Such  a  result  indicates  that  informative  EEG  power  at  parietal  and  occipital  head  locations 
covaried  systematically  with  the  spatial  location  of  covert  attention.  This  pattern  within  the 
covariance  matrices  was  common  to  all  good  performing  subjects  while  poorer  performing 
subjects  displayed  relatively  uniform  errors  (figure  19). 
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LDA  Random  Forests 

Figure  18:  Confusion  matrices  from  the  best-performing  subject  (10)  in  the  horizontal  condition 

The  confusion  matrices  indicate  the  probability  of  a  model  to  predict  one  of  the  n-bins  given  the  actual  attention 
location  in  the  held-out  test  data.  The  actual  attention  location  bins  are  ordered  along  the  rows  from  top,  the  left 
most  attention  bin,  to  bottom,  the  right  most  attention  bin,  while  probability  of  predicting  a  given  bin  along  the 
columns  from  left  most  bin  to  right  most  bin.  Error  free  models  would  display  perfect  diagonal  matrix  structure. 
Despite  neither  the  LDA  nor  Random  Forest  having  the  attention  bin  location  ordinarily  built  into  their  model  fitting, 
the  best-performing  subject's  confusion  matrices  display  block-diagonal  structure.  This  indicates  that  well  fit 
models  of  good  performers  most  often  make  prediction  errors  into  bins  neighboring  the  correct  attention  location. 
Such  a  result  implies  shifting  locus  of  attention  and  occipito-parietal  alpha  and  beta  power  covary  in  a  linearly 
consistent  fashion  in  high  performing  subjects. _ 
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LDA  Random  Forests 

Figure  19:  Confusion  matrices  of  the  worst-performing  subject  in  the  horizontal  condition 

Confusion  matrices  of  poorer  performing  subjects  display  uniform  distributions  of  probability  to  predict  attention 
locations  given  an  actual  attention  location. _ 


Vertical  Tracking 


Tracking  of  attention  was  successful  overall  when  attention  varied  in  the  vertical 
direction.  However,  accuracy  rates  are  systematically  lower  than  in  the  horizontal  tracking 
case.  All  but  three  of  the  subjects  attention  could  be  tracked  at  a  rate  significantly  greater  than 
chance  at  all  degrees  of  granularity,  while  three  of  the  11  subjects  posterior  predictive  accuracy 
rates  were  at  least  15%  greater  than  chance  at  all  granularities.  The  best-performing  subjects 
expected  accuracy  rates  were  61.6%,  46.1%,  30.3%  and  20.8%  when  predicting  into  two,  three. 


44 


five  or  seven  bins,  respectively.  Again,  linear  models  generally  performed  better  though  the 


difference  was  even  less  dramatic  than  in  the  horizontal  case. 
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Figure  20:  Tracking  accuracy  rates  from  the  vertical  condition  at  increased  discriminant  precision 

Empirical,  expected  predictive  accuracy  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear 
Random  Forest  models  (bottom)  as  the  vertical  attention  target  sample  space  was  divided  into  more  numerous 
and  narrow  bins  (as  shown  in  figure  13).  Chance  accuracy  at  each  number  of  bins  displayed  as  the  bold,  black  line. 
All  but  a  few  subjects  performed  significantly  better  than  chance  up  to  and  including  7  vertical  bins.  Linear 
Models,  on  average,  performed  better  than  nonlinear  models. 
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Maximal  communication  speed  was  seen  in  the  three-bin  case  for  many  subjects.  The 


highest  overall  ITR  of  7.42  was  found  in  the  three-bin  case  of  subject  11.  ITR,  like  tracking 
accuracy,  was  lower  than  that  found  while  tracking  horizontally  varying  attention. 
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Figure  21:  ITRs  from  the  vertical  condition  at  increased  discriminant  precision 

Empirical,  expected  ITRs  and  97.5%  Cl  from  Linear  Discriminant  Analysis  models  (top)  and  nonlinear  Random 
Forest  models  (bottom)  as  the  vertical  tracking  was  performed  with  greater  granularity  (as  shown  in  figure  13). 
The  maximal  ITR  is  seen  in  the  3-bin  case  for  subject  11.  The  benefit  of  tracking  with  more  precision  than  just 
discriminating  vertical  hemifield  of  attention  locus  can  be  clearly  seen. 
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Topographic  visualization  of  models  in  the  three-bin  case  indicate  that  alpha,  low-beta 


and  high-beta  EEG  power  varied  most  significantly  in  the  very  focused  region  of  occipito¬ 
parietal  junction  over  the  central  sulcus,  while  gamma  band  power  showed  no  discerned 
differences  between  vertical  attention  locations  (figure  22).  Much  more  activity  is  seen  in  these 
bands  and  over  this  region  when  attention  was  directed  above  fixation,  while  much  less  activity 
is  seen  in  this  region  when  attention  is  directed  below  fixation.  When  attention  was  directed 
near  fixation,  activity  was  at  a  median  value  between  the  higher  activity  seen  when  attending 
upwards  and  the  lower  levels  of  activity  when  attending  downwards.  Random  Forest  feature 
importance  indicated  that  low-beta  activity  over  this  central,  occipito-parietal  region  was  by  far 
the  most  important  feature  contributing  to  correct  tracking  of  covert  attention  locus  in  the 
vertical  direction. 
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RF  all  bins 

Figure  22:  Models  from  the  best-performing  subject  (11)  in  the  vertical  condition 

Left:  LDA  model  class  means  for  the  3  bin  case.  Class  means  are,  from  left  to  right,  attention  up,  attention 
ahead  and  attention  down.  Increased  synchrony  can  be  seen  in  red  while  desynchrony  can  be  seen  in  blue. 
Attention  directed  above  fixation  is  associated  with  increased  synchrony  centrally  over  occipito-parietal 
sulcus  and  the  lateral  fissure  in  alpha,  low-beta  and  high-beta  bands,  while  attention  below  fixation  is 
associated  with  desynchrony  in  this  region.  Attention  near  fixation  is  associated  with  average  synchrony 
values  between  attention  up  and  attention  down. 

Right:  RF  model  feature  importance  supports  the  importance  of  this  region,  within  these  EEG  bands,  for 
correct  prediction  of  covert  attention  locus. 
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RF  all  bins 

Figure  23:  Models  from  the  worst  performing  subject  (5)  in  the  vertical  condition 

Models  fit  to  data  from  subjects  with  poorer  predictive  performance  do  not  emphasize  the  central 
occipito-parietal  junction  region  in  the  alpha,  low-beta  and  high-beta  bands. 
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Unlike  in  the  horizontal  case,  confusion  matrix  analysis  shows  that  tracking  attention 


vertically  is  a  more  difficult  prospect  (figures  24  &  25).  Models  showed  a  strong  preference  to 
predict  attention  in  the  upward  direction.  In  conjunction  with  the  analysis  of  spectral-spatial 
activity  of  EEG  contributing  to  accurate  prediction,  it  is  likely  that  increased  activity  in  alpha, 
and  beta  bands  was  highly  indicative  of  attention  being  towards  the  extreme  upper  region  of 
the  visual  field,  while  the  characteristic  features  of  EEG  power  while  attention  is  directed 
anywhere  else  along  elevation  was  more  difficult  to  discern. 


LDA  Random  Forests 

Figure  24:  Confusion  matrices  from  the  best-performing  subject  (11)  in  the  vertical  condition 

Confusion  matrices  for  the  vertical  condition  are  displayed  with  actual  attention  location  along  rows,  ordered  from 
most  top  bin  to  most  bottom  bin,  with  probability  of  predicting  a  given  bin  along  columns  ordered  from  most  top 
(left)  to  most  bottom  (right). 

Confusion  matrices  for  all  good  performing  subjects  display  some  block-diagonal  structure,  but  also  indicate  a 
strong  preference  for  prediction  of  the  top  most  attention  location. 
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LDA  Random  Forests 

Figure  25:  Confusion  matrices  from  the  worst-performing  subject  in  the  vertical  condition 

Confusion  matrices  of  poorer  performing  subjects  display  a  uniform  distribution  for  prediction  of  any  attention 
location  given  an  actual  attention  location. _ 
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Discussion 


Experimental  results  show  that  the  locus  of  visual  attention  can  be  tracked  in  one 
dimension  using  EEG.  Tracking  was  successful  at  varying  levels  of  spatial  precision  when 
attention  location  varied  horizontally  or  vertically.  The  most  informative  EEG  features  were 
found  in  the  low-beta  band,  between  16  and  20  Hz,  over  the  parietal  and  occipital  lobes. 
Additionally,  alpha  and  higher-beta  band  activity  in  parietal  and  occipital  lobes  contributed  to 
increased  performance  of  tracking. 

Lateralized  occipital  and  parietal  EEG  features  in  alpha  and  beta  bands  were  most 
informative  when  tracking  attention  to  targets  with  positions  that  varied  horizontally.  These 
results  agree  with  earlier  reports  that  link  lateralized  alpha  and  beta  band  activity  to  the  side- 
to-side  direction  of  visual  attention  (Bahramisharif  2011,  Gerven  et  al.  2013,  Kelly  et  al.  2006, 
Mangun  1995,  Ricco  et  al.  2012,  Thorpe  et  al.  2012,  Tonin  et  al.  2012  &  2013).  EEG  activity  in 
the  alpha  and  beta  bands  of  the  central,  occipito-parietal  junction,  over  the  central  sulcus,  were 
found  to  be  most  associated  with  attention  varying  in  the  vertical  direction.  This  result  has  not 
yet  been  reported  in  EEG  literature,  but  it  agree  spatially  with  magnetoencephalography  (MEG) 
and  functional  magnetic  resonance  imaging  (fMRI)  studies  of  directed  visual  attention 
(Bahramisharif  2010  &  2011,  Silver  2005). 

Tracking  accuracy  rates  found  when  parsing  the  visual  space  into  two  bins  were 
comparable  to  those  reported  for  other  reported  covert  attention  EEG  BCI  paradigms  (Tonin  et 
al.  2012&  2013).  For  this  set  of  EEG  features,  regularized-Bayesian  Linear  Discriminant  Analysis 
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outperformed  nonlinear  Random  Forest  models  in  all  but  a  few  cases.  This  indicates  that 
simpler  models  are  likely  sufficient  to  track  attention  when  using  CSPs. 

The  fastest  reported  communication  rates  of  EEG  BCIs,  which  use  endogenous  attention 
signals,  fall  just  above  eight  BPM  with  a  performance  rate  of  85%  in  the  best  reported  subjects, 
2.85  seconds  of  EEG  collection  per  trial  and  a  binary  left  versus  right  attention  choice  (Tonin  et 
al.  2012  &  2013).  Covert  attention  BCIs  of  this  type  are  limited  in  their  ITR  by  the  binary  nature 
of  their  attention  location  choice.  In  this  experiment,  two  subjects  produced  ITRs  in  excess  of 
eight  BPM  in  the  three  bin  case  while  tracking  attention  that  varied  horizontally.  The  best  ITR 
seen  in  this  experiment,  11.35,  was  seen  while  tracking  horizontally  varying  attention  into  two 
bins.  The  best  communication  rate  seen  while  tracking  attention  vertically  was  7.42  BPM,  while 
tracking  attention  into  three  vertical  bins. 

Qualitatively,  subjects  could  be  grouped  into  high  performers,  or  those  whose  attention 
tracking  accuracy  rates  were  consistently  above  chance,  variable  subjects  whose  accuracy  rates 
varied  depending  on  orientation  of  tracking,  and  poor  performers  whose  performance  was 
consistently  near  or  at  chance  accuracy  rates.  During  experimentation,  some  of  the  variable 
subjects  reported  difficulty  directing  covert  visual-spatial  attention  in  one  direction  or  the 
other.  These  difficulties  were  subject-specific  and  coincide  with  reduced  performance  in  the 
orientation  reported  as  difficult  for  that  subject.  Performance  of  this  attention  tracking  method 
therefore  is  influenced  by  both  detectable  EEG  activity  as  well  as  a  subject's  proficiency  in 
directing  gaze-independent,  covert  visual-spatial  attention. 
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The  success  of  this  method  encourages  further  research  on  the  EEG  based  tracking  of 


bottom-up  covert  visual-spatial  attention  for  BCI  use.  Specifically,  tracking  of  attention  in  two 
dimensions  simultaneously  should  be  attempted. 
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IV.  Tracking  attention  in  two  dimensions  using  EEG 


Introduction 

The  results  from  experiment  two  showed  that  spatial  locus  of  covert,  visual  attention 
could  be  effectively  tracked  using  EEG  signals  while  attention  was  being  directed  to  targets  as 
they  appeared.  Tracking  was  successful  while  attention  varied  in  one  dimension,  either  the 
horizontal  direction  or  in  the  vertical  direction.  The  desirable  next  step  for  this  research  would 
be  to  track  attention  as  it  varies  in  more  than  one  dimension.  Silver  has  reported  topographic 
maps  of  two  dimensionally  varying  covert  attention  acquired  by  fMRI  (Silver  2005),  while 
Bahramisharif  and  colleagues  have  use  MEG  detect  the  horizontal  and  vertical  orientation  of 
covet  attention  and  to  predict  the  continuous,  radial  orientation  of  attention  (Bahramisharif  et 
al.  2010  and  Gerven  et  al.  2009).  However,  no  studies  have  reported  characterizing  EEG  activity 
associated  with  directed,  covet  attention  as  it  varies  in  two  dimensions. 

The  present  experiment  uses  much  of  the  methods  described  in  the  previous  two 
chapters,  but  requires  the  subjects  to  vary  he  locus  of  attention  in  two  spatial  dimensions 
simultaneously.  Utilizing  the  informative  EEG  power-band  features  identified  in  the  previous 
chapter,  predictive  models  will  again  be  used  to  both  characterize  EEG  activity  associated  with 
directed  visual  attention  and  evaluate  the  precision  to  which  locus  of  attention  can  be  reliably 
tracked. 
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Methods 


The  environment,  tools,  subjects  and  experimental  procedures  used  in  this  experiment 
are  identical  to  those  described  in  experiment  two  except  for  the  following  differences. 
Attention  target  locations  were  sampled  uniformly  from  the  34°  by  34°  square  area  centered 
on  the  fixation  point  (figure  26,  top).  Targets  were  again  placed  at  a  minimum  distance  of  2° 
from  the  preceding  target  location.  A  total  of  5040  trials  were  collected  from  12  sessions,  each 
containing  420  trials,  over  the  course  of  two  or  three  days. 


Figure  26:  Experiment  screen  and  prediction  bins 

Top:  The  display  is  identical  to  experiment  2.  In  the  displayed  trial,  the  letter  H  is  the  covert  attention  target. 
Target  locations  were  sampled  uniformly  in  the  square  defined  by  [—17°,  17°]  at  top-left  and  [17°,  — 17°]  at 
bottom-right. 

Bottom:  The  attention  target  sample  space  divided  into  four  equal  sized  bins  for  classification.  Classification  was 
also  performed  using  nine  (three  x  three)  bins,  25  (five  x  five)  bins  and  49  (seven  x  seven)  bins. 
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Signal  processing  and  feature  extraction  methods  were  identical  to  those  described  in 


experiment  two. 

As  in  experiment  two,  LDA  and  RF  were  used  for  predictive  modeling.  Classification 
targets  were  defined  by  binning  the  attention  target  sampling  space  on  one  of  two  ways.  First, 
the  target  bins  were  defined  as  two  dimensional  squares  numbering  either  four  (two  x  two), 
nine  (three  x  three),  25  (five  x  five)  or  49  (seven  x  seven),  with  the  sides  of  each  bin  being  of 
length  17°,  11°,  6°  and  4°,  respectively  (figure  26,  bottom).  Second,  two  dimensionally  varying 
attention  was  tracked  in  one  dimension  at  a  time,  in  an  identical  manner  as  experiment  two. 
Predictive  accuracy,  ITR,  topographies  and  confusion  matrices  were  computed  in  the  same 
fashion  as  in  experiment  two. 


Results 


Two  dimensional  variation  in  locus  of  attention 


Tracking  of  attention  in  two  dimensions  was  successful  when  the  attention  space  was 
divided  into  four  or  seven  equal-sized  bins  (figure  26).  When  tracking  attention  in  two 
dimensions,  at  these  levels  of  precision,  accuracy  rates  were  significantly  above  chance  for  all 
subjects.  In  the  four  bin  case  eight  of  the  11  subjects  posterior  accuracy  rate  lower  bound  was 
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at  least  15%  above  chance.  However,  rates  of  accuracy  varied  widely  from  37.8%  to  28.1%  as 


compared  to  chance  of  25%  when  predicting  one  of  four  bins  of  attention,  while  accuracy 
ranged  from  19.3%  to  12.7%  as  compared  chance  of  11.1%  when  predicting  into  one  of  nine 
equal-sized  bins.  More  precise  prediction  resulted  in  accuracy  rates  barely  above  chance  or  not 
significantly  above  chance. 
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Figure  27:  Accuracies  when  tracking  attention  in  two  dimensions 

Empirical,  expected  predictive  accuracy  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear 
Random  Forest  models  (bottom)  as  the  two  dimensional  attention  target  sample  space  was  divided  into  more 
numerous,  square  bins  (as  shown  in  figure  2).  Chance  accuracy  at  each  number  of  bins  displayed  as  the  bold,  black 
line.  Linear  models  significantly  outperform  chance  prediction  rate  for  all  subjects  in  up  to  9,  3  horizontal  by  3 
vertical,  bins.  In  the  case  of  some  subjects,  correct  attention  bin  could  be  predicted  at  significantly  better  than 
chance  rate  from  among  49  different,  equal-sized  bins.  As  compared  to  linear  models,  nonlinear  models  performed 
more  consistently  across  subjects  and  generally  better  when  trying  to  predict  attention  location  from  among  25  or 
49  bins. 
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ITRs  produced  from  linear  models  consistently  outperformed  nonlinear  models  when 


tracking  attention  in  two  dimensions  (figure  28).  Additionally,  the  increased  tracking  accuracy 
seen  in  the  four-bin  case,  (two  by  two),  produced  the  greatest  ITRs.  The  highest 
communication  seen  for  two  dimensional  tracking  was  6.65  BPM  in  the  four-bin  case  from 
subject  11. 
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Figure  28:  ITRs  when  tracking  attention  in  two  dimensions 

The  majority  of  subjects  produced  maximal  ITR  in  the  four  bin  case  (2x2).  Increasing  the  precision  to  which  attention 
locus  is  tracked  does  not  aid  in  ITR  when  tracking  attention  in  two  dimensions. 
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Visualizing  the  four-way  models  again  showed  parietal  and  occipital  regions  in  EEG 


alpha,  low-beta  and  high-beta  bands  to  be  most  indicative  of  locus  covert  attention  when  it 
varied  in  two  dimensions  (figure  29).  Within  the  linear  model  class  means,  diagonally-opposite 
classes  displayed  identical  spatial  patterns  with  opposite  polarity  in  these  bands.  Generally,  for 
the  two  attention-up  classes,  increased  activity  was  seen  in  left  occipito-parietal  regions  and 
decreased  activity  was  seen  in  right  occipito-parietal  regions  relative  to  the  two  attention-down 
classes.  The  similarity  of  spatial  patterns  with  opposite  polarity  is  likely  a  result  of  the  shared 
covariance  structure  between  the  classes  of  the  linear  discriminant  models,  and  inference  from 
these  models  upon  EEG  recordable  neural  activity  associated  with  two  dimensionally  varying 
attention  should  be  discouraged.  The  nonlinear  Random  Forest  models,  although  with  slightly 
lower  overall  performance,  show  the  central  occipito-parietal  region  of  the  low-beta  band  to  be 
most  useful  in  accurate  tracking,  followed  by  right  parietal  regions  in  alpha  and  high-beta 
bands. 


62 


oooooooo 


00000000 

•  •  •  • 

Alpha  band  Low-Beta  band  High-Beta  band  Gamma  band 

Figure  29:  Models  from  the  best  performing  subject  (11)  when  tracking  attention  in  two  dimensions 
Top:  LDA  models  fit  when  attention  is  varied  in  two  dimensions  indicate  activation  in  the  occipito-parietal  regions 
seen  in  models  fit  to  the  horizontal  and  vertically  varying  attention  conditions.  Again,  models  emphasize  activity 
primarily  in  alpha,  low-beta  and  high-beta  bands. 

Bottom:  RF  model  feature  importance  indicates  that  alpha  activity  in  right  parietal  and  low-beta  activity  in  central 
occipito-parietal  junction  contribute  most  to  correct  prediction  of  covert  attention  locus. 
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Tracking  horizontally  in  the  while  attention  varied  in  two  dimensions 


When  tracking  the  horizontal  position  of  attention  while  the  locus  of  attention  varied  in 
two  spatial  dimensions,  accuracy  decreased  in  some  cases  and  increased  in  other  when 
compared  to  tracking  in  cases  where  locus  of  attention  varied  only  in  the  horizontal  direction 
(figures  14,  30).  Tracking  accuracy  for  all  subjects  exceeded  chance,  in  statistical  significance,  at 
all  levels  of  precision  when  tracking  was  conducted  with  linear  models,  while  two  failed  to 
exceed  chance  accuracy  when  tracking  was  done  with  nonlinear  models.  In  the  two-bin  case, 
seven  of  11  subjects  posterior  accuracy  lower-bound  exceeded  chance  by  15%  or  more. 
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Figure  30:  Accuracy  rates  tracking  horizontally  when  attention  varies  in  two  dimensions 

Empirical,  expected  predictive  accuracy  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear 
Random  Forest  models  (bottom)  as  the  two  dimensional  attention  target  sample  space  was  divided  into  vertical 
bins  similar  to  that  used  to  analyze  the  horizontal  condition  (figure  1).  Chance  accuracy  at  each  number  of  bins 
displayed  as  the  bold,  black  line.  Linear  models  significantly  outperform  chance  prediction  rate  for  all  subjects  in 
up  to  seven  bins.  Nonlinear  model  predictions  were  less  accurate  than  linear  model  predictions  in  all  subjects 
except  subject  3.  Subject  3's  locus  of  attention  could  be  predicted  with  significantly  more  accuracy  in  the  five  and 
seven  bin  case  with  using  nonlinear  models. 
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The  pattern  of  communication  rates  followed  that  seen  in  tracking  horizontally  varying 


attention,  with  a  reduction  in  performance  (figuresl5,  31).  Subject  5  performed  exceptionally 
well  when  Random  Forests  were  used  to  track  their  attention  in  the  five-bin  and  seven-bin 
cases,  and  as  a  result  the  communication  rates  were  6.59  BPM  and  11.18  BPM.  These 
improved  ITRs  display  the  benefit  of  attempting  to  track  locus  of  attention  more  precisely. 
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Figure  31:  ITRs  tracking  horizontally  when  attention  varies  in  two  dimensions 

ITRs  for  tracking  locus  of  attention  horizontally  while  it  varies  in  two  dimensions  are  below  that  seen  when  tracking 
attention  that  varies  only  horizontally.  For  most  subjects,  maximal  ITR  is  seen  in  the  two-bin  case,  indicating 
increased  granularity  of  tracking  does  not  yield  increased  communication  speed. 
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Visualization  of  the  horizontal  tracking  models  of  two  dimensionally  varying  covert 


visual  spatial  attention,  in  the  three-bin  case,  revealed  activation  patterns  similar  to  attention 
tracking  when  attention  varied  only  in  the  horizontal  dimension  (figures  16,  32).  Attention 
towards  the  left  or  the  right  visual  fields  was  associated  with  relatively  more  activation 
ipsilaterally  and  less  activation  contralaterally  in  occipito-parietal  alpha,  and  beta  bands  while 
attention  near  fixation  was  associated  with  relatively  less  alpha,  and  low-beta  and  high-beta 
band  power  bilaterally  in  occipital  and  parietal  lobes.  Random  Forest  model  feature 
importance  also  found  the  lateral  parietal  regions  of  alpha  and  beta  bands  to  be  most 
important  for  correct  attention  locus  tracking,  with  changes  in  low-beta  activity  in  lateral 
parietal  lobe  to  be  the  most  crucial  features  in  correct  prediction. 
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RF  all  bins 

Figure  32:  Models  from  the  best-performing  subject  (10)  when  tracking  horizontally  while  attention 
varies  in  two  dimensions 

Results  of  analysis  of  vertical  margins  for  the  two  dimensional  condition  are  comparable  to  the  horizontal 
condition.  LDA  models  (left)  display  ipsilateral  synchrony  as  well  as  contralateral  desynchrony  in  alpha, 
and  beta  bands  while  attention  near  the  horizontal  position  of  fixation  display  bilateral  desynchrony. 
Additionally,  RF  models  indicate  parietal  and  occipital  alpha  and  low-beta  bands  contribute  most  to 
accurate  prediction  attention  locus. 
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Confusion  matrices  for  the  horizontal  tracking  of  two  dimensionally  varying  covert 


spatial  attention  also  displayed  the  near-block-diagonal  structure  seen  in  horizontal  tracking  of 
one  dimensionally  varying  attention  (figures  8,  24). 


LDA  Random  Forests 

Figure  33:  Confusion  matrices  from  the  best-performing  subject  (10)  when  tracking  horizontally  while 
attention  varies  in  two  dimensions 

When  predicting  the  vertical  marginal  bin  of  attention  location  when  locus  varied  in  two  dimensions, 
confusion  matrices  again  display  a  block-diagonal  structure  similar  to  horizontal  condition  (figure  8). 
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Tracking  vertically  while  attention  varied  in  two  dimensions 


When  tracking  locus  of  covert  visual  attention  vertically  while  subject's  attention  varied 
in  two  dimensions,  the  same  pattern  of  reduced  accuracy  in  the  best  performers  and  increased 
accuracy  in  the  worst  performers  can  be  seen  (figures  20,  36).  Only  one  subject's  performance 
consistently  failed  to  exceed  chance,  while  a  second  subject's  performance  decreased  below 
chance  when  prediction  was  made  into  five  or  seven  bins  of  the  vertical  sample  space.  In  most 
cases,  performance  was  below  that  of  horizontal  attention  tracking  under  the  two  dimensional 
conditions.  In  the  two  bin  case,  four  of  the  11  subject's  posterior  accuracy  rate  lower-bound 
exceeded  chance  by  at  least  15%. 
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Figure  34:  Accuracy  rates  tracking  vertically  when  attention  varies  in  two  dimensions 

Empirical,  expected  predictive  accuracy  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear 
Random  Forest  models  (bottom)  as  the  two  dimensional  attention  target  sample  space  was  divided  into  horizontal 
bins  similar  to  that  used  to  analyze  the  vertical  condition  (as  shown  in  figure  1).  Chance  accuracy  at  each  number 
of  bins  displayed  as  the  bold,  black  line.  Linear  models  significantly  outperform  chance  prediction  rate  for  all 
subjects  in  up  to  seven  vertical  bins.  In  most  cases  linear  models  outperformed  nonlinear  models. 
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Communication  rates  for  tracking  attention  vertically  while  it  varied  in  two  dimensions 


can  be  seen  in  figure  35.  Like  in  the  horizontal  tracking  of  two  dimensionally  varying  attention, 
one  subject's  accuracy  rate  declined  more  gradually  as  tracking  was  performed  more  precisely. 
This  subject's  ITR  increased  in  the  three-bin  to  seven-bin  cases.  This  subject  producing  a 
maximal  ITR  of  9.37  BPM  when  linear  models  discriminated  between  seven  bins  of  attention 
location  along  elevation.  The  remainder  of  the  subjects  produced  maximal  ITRs  in  either  the 
two-bin  or  three-bin  cases,  and  these  ITRs  were  generally  lower  than  those  seen  when  tracking 
attention  varying  only  in  the  vertical  direction. 
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Figure  35:  ITRs  tracking  vertically  while  attention  varies  in  two  dimensions 

Empirical,  expected  ITR  and  97.5%  Cl  of  Linear  Discriminant  Analysis  models  (top)  and  nonlinear  Random  Forest 
models  (bottom).  ITR  is  less  than  the  comparable  rates  when  attention  is  varied  only  in  the  vertical  direction. 
The  increasing  ITR  of  subject  7  nicely  demonstrates  that  when  discriminating  attention  locations  between  more 
numerous  bins,  mild  increases  in  predictive  accuracy  rate  results  in  a  large  increase  in  communication  speed. _ 
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Topographic  visualization  of  these  models  showed  very  similar  activity  patterns  to  one 


dimensional  vertical  tracking.  The  most  significant  features,  and  those  most  important  to 
successful  tracking,  varied  in  the  very  concentrated  region  of  the  occipito-parietal  junction  over 
or  near  the  central  sulcus  in  the  alpha,  and  beta  bands  (figures  22,  36).  In  the  case  of  vertical 
tracking  while  attention  varied  in  two  dimensions,  the  low-beta  band,  and  to  a  lesser  degree 
the  high-beta  band  contributed  most  to  successful  tracking  when  using  Random  Forests,  while 
variance  in  alpha  activity  had  much  less  effect  on  performance. 
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Figure  36:  Models  from  the  best-performing  subject  (10)  when  tracking  vertically  while  attention  varies 
in  two  dimensions 

Results  of  analysis  of  horizontal  margins  for  the  two  dimensional  condition  are  comparable  to  the  vertical 
condition.  Again,  LDA  models  (left)  of  attention  up  display  synchrony  in  the  occipito-parietal  junction 
above  the  lateral  fissure  in  alpha,  and  beta  bands  while  attention  down  shows  desynchrony  in  this  region. 
RF  model  feature  importance  indicates  low  and  high-beta  in  central  occipito-parietal  junction  provides  the 
most  informative  features  for  correct  prediction  of  attention  locus. 
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Confusion  matrices  for  both  linear  and  nonlinear  models  displayed  a  strong  preference 


for  the  maximally  upper,  central  and  maximally  lower  bins  of  the  vertical  attention  space  giving 
rise  to  the  vertically  stripped  pattern  seen  (figure  37). 


LDA  Random  Forests 

Figure  37:  Confusion  matrices  from  the  best-performing  subject  (11)  when  tracking  vertically  while  attention 
varies  in  two  dimensions 

When  predicting  the  horizontal  marginal  bin  of  attention  when  attention  locus  varied  in  two  dimensions,  confusion 
matrices  display  a  preference  for  predicting  attention  is  either  near  fixation  (central  column)  or  at  the  extreme  top 
or  bottom  of  the  visual  field.  Such  a  preference  indicates  model  difficulty  discriminating  locus  of  covert  attention  in 
the  vertical  direction  unless  it  is  very  near  or  very  far  away  from  fixation. 
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Discussion 


Using  the  methods  described,  attention  could  be  tracked  in  two  dimensions  using  EEG, 
at  rates  significantly  above  chance  in  up  to  49  (seven  x  seven)  distinct  regions  of  the  visual  field, 
for  all  but  a  few  of  the  subjects  tested.  ITRs  reported  for  the  four-bin  case  of  two  dimensional 
attention  tracking  approach  the  best  reported  communication  rates  seen  from  EEG  BCI 
literature  utilizing  endogenous,  top-down  attention  signals  (Tonin  et  al.  2012&  2013).  Though 
accuracy  rates  were  above  chance,  the  rates  are  by  no  means  high  enough  to  be  used  in  an 
online  BCI.  However,  these  positive  results  encourage  further  research  towards  using 
endogenous  brain-activity  associated  with  attention  directed  at  non-specific  targets  for  BCI 
control.  These  signals  can  be  used  to  predict  attention  locus  more  precisely  than  in  the  binary 
fashion  used  so  far  in  EEG  BCI  research.  More  precise  prediction  of  attention  location  will  lead 
to  many  fold  increases  in  communication  rates  once  accuracy  rates  can  be  improved. 

A  combination  of  the  important  EEG  features  found  for  horizontal  and  vertical  tracking 
contributed  to  successful  tracking  of  attention  in  two  dimensions.  Variation  in  lower-beta  band 
activity  centrally  located  at  the  occipito-parietal  junction,  over  the  central  sulcus,  contributes 
most  to  successful  tracking.  Additionally,  right  lateralized,  parietal  alpha  and  low  beta  activity 
provide  additional  information  regarding  position  of  attention.  These  results  are  novel,  as 
there  is  no  literature  reporting  the  EEG  activity  associated  with  directed,  bottom-up  covert 
attention,  in  one  or  two  dimensions.  However,  previous  MEG  and  fMRI  literature  regarding 
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two  dimensional,  top-down  directed,  covert  attention  agree  spatially  with  these  results 
(Bahramisharif  2010  &  2011,  Silver  2005). 

Tracking  of  attention  in  one  dimension  while  attention  varied  in  two  dimensions  also 
proved  successful.  Regions  and  EEG  bands  implicated  by  predictive  modeling  agree  with  the 
results  from  experiment  two,  where  attention  varied  in  only  one  dimension.  Accuracy  rates, 
and  resultantly  ITR,  are  not-surprisingly  reduced  compared  to  when  attention  varied  in  only  one 
dimension.  Such  a  reduction  in  accuracy  must  be  the  consequence  of  brain  activity  associated 
with  variation  in  attention  position  in  the  untracked  dimension  introducing  irreducible  noise  in 
the  EEG  feature  set  used  for  tracking.  Despite  the  reduced  accuracy,  tracking  attention  on  one 
dimension  while  allowing  subjects  to  vary  attention  location  in  two  dimensions  is  more  directly 
useful  as  a  passive  BCI.  Such  a  BCI  could  be  immediately  used  to  track  attention  locus  to 
vertical  or  horizontal  bands  while  a  subject  performed  any  fixed-gaze  task. 
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V.  Conclusion 


Experimental  results  show  that  the  locus  of  visual  attention  can  be  tracked  using  EEG. 
Attempts  to  track  attention  using  SSVEP-like  signals  induced  from  peripheral,  unattended 
flickering  stimuli  in  experiment  one  were  not  successful.  The  highly  salient,  flickering  stimuli 
were  effectively  suppressed  while  attention  was  directed  at  intermediate  locations,  and 
induced  no  detectable  SSVEP-like  signal.  However,  despite  the  failure  of  this  method  to 
produce  SSVEPs,  tracking  was  still  successful  when  using  endogenous  alpha  and  beta  band 
power  lateralizations  associated  with  directed,  covert  attention. 

Tracking  was  successful  at  varying  levels  of  spatial  precision  when  attention  location 
varied  horizontally,  vertically  or  in  two  spatial  dimensions.  The  most  informative  EEG  features 
were  found  in  the  low-beta  band,  between  16  and  20  Hz,  over  the  parietal  and  occipital  lobes. 
Additionally,  alpha  and  higher  beta  band  activity  in  parietal  and  occipital  lobes  contributed  to 
increased  performance  of  tracking. 

Qualitatively,  subjects  could  be  grouped  into  high-performers,  or  those  whose  attention 
tracking  accuracy  rates  were  consistently  above  chance,  variable  subjects  whose  accuracy  rates 
varied  depending  on  orientation  of  tracking,  and  poor  performers  whose  performance  was 
consistently  near  or  at  chance  accuracy  rates.  During  experimentation,  some  of  the  variable 
subjects  reported  difficulty  directing  covert  visual-spatial  attention  in  one  direction  or  the 
other.  These  difficulties  were  subject-specific  and  coincide  with  reduced  performance  in  the 
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orientation  reported  as  difficult  for  that  subject.  Performance  of  this  attention  tracking  method 


therefore  is  influenced  by  both  detectable  EEG  activity  as  well  as  a  subject's  proficiency  in 
directing  gaze-independent,  covert  visual-spatial  attention. 

Evaluating  the  performance  in  tracking  the  locus  of  attention  at  increased  degrees  of 
precision  is  an  attempt  to  approach  the  goal  of  continuously  tracking  the  position  of  covert, 
visual-spatial  attention  in  two  dimensions.  This  goal  would  allow  for  the  use  of  covert  attention 
as  a  cursor-like  control  signal  for  EEG  BCIs.  Such  a  control  method  would  allow  or  fully 
functional  interfaces  for  BCI  users  like  those  found  in  modern,  user-friendly  computer 
interfaces.  In  experiment  three,  tracking  the  one  dimensional  position  of  bottom-up  attention 
could  be  accomplished  at  rates  significantly  better  than  chance  in  the  majority  of  subjects  in  up 
to  seven  discrete  bins  of  visual  space.  Tracking  the  two  dimensional  position  of  bottom-up 
covert,  visual-spatial  attention  was  also  at  above  chance  rates. 

The  success  of  this  method  encourages  further  research  on  the  EEG  based  tracking  of 
bottom-up  covert  visual-spatial  attention  for  BCI  use.  Tracking  of  two  dimensionally  varying 
attention  in  a  single  dimension  in  a  passive  manner,  as  shown  in  experiment  three,  could  easily 
be  implemented  in  an  online  EEG  BCI,  as  is,  to  detect  attention  locus  relative  to  gaze. 
Additionally,  further  research  on  method  should  involve  experimental  displays  with  more  rich 
backgrounds  such  as  distracter  targets  or  live  scenes  to  evaluate  tracking  in  a  more  complex 
environment.  Improvements  in  EEG  feature  extraction  and  predictive  model  building  to 
increase  accuracy  should  also  be  goals  for  further  research.  With  improved  features,  tracking 
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of  attention  locus  should  increase  in  granularity  towards  the  goal  of  predictive  regression 
modeling  of  absolute  position  of  bottom-up  covert  visual  attention. 
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